Build: keep CI Docker images dependency-only - #7755
Conversation
The devcontainer workflow rebuilds and pushes abacus-{gnu,cuda}:latest
on every push to develop, and the Dockerfiles contained a cache-busting
ADD of the develop ref plus a full in-image ABACUS build. The published
image therefore changed after EVERY merge, so self-hosted runners had to
re-download gigabytes of image layers at job start - the 'Initialize
containers' step took 8-78 min in CI (measured via the GitHub API).
The CI never uses the baked-in binary: every workflow checks out the PR
code and rebuilds from source. Removing the cache-bust and the in-image
build makes the image content change only when dependencies themselves
change, so runners can keep it cached for weeks.
Note: users who ran 'docker run' to get a prebuilt ABACUS from these
images should instead build from source (docs/quick_start) or use a
release binary.
|
Sorry but, isn't the docker images distributed to downstream users? CC: @ZhouXY-PKU |
|
Sorry, but these dockerfiles are designed to provide ABACUS that can be executed directly... |
mohanchen
added
the
Compile & CICD & Docs & Dependencies
|
I understand the goal of stabilizing the CI images, but abacus-gnu is still used by performance.yml and downstream users as a directly runnable image, so removing the binary would be a regression. Please preserve the existing image contract and publish a separate, versioned CI dependency image for CI use, with an actual image build test. |
|
Moreover, please note, that most of the image download overhead comes from the Intel or CUDA toolchains, which can easily amount to 3~4 GiB, rather than from ABACUS itself. Therefore, this change would not even address the main source of the problem it is intended to solve at all... |
|
That being said, I agree the current Dockerfile layout is a bit tricky to handle with. Since I'm not a fan of Dockerfile distribution, I'm not too convinced how it can be optimized; it might need to be left discussion. If I was to handle CI testings, I would re-design a dedicated Docker workflow to cover everything (toolchain, building, and testing) at once. |
5 participants
The devcontainer workflow rebuilds and pushes abacus-{gnu,cuda}:latest on every push to develop, and the Dockerfiles contained a cache-busting ADD of the develop ref plus a full in-image ABACUS build. The published image therefore changed after EVERY merge, so self-hosted runners had to re-download gigabytes of image layers at job start - the 'Initialize containers' step took 8-78 min in CI (measured via the GitHub API).
The CI never uses the baked-in binary: every workflow checks out the PR code and rebuilds from source. Removing the cache-bust and the in-image build makes the image content change only when dependencies themselves change, so runners can keep it cached for weeks.
Note: users who ran 'docker run' to get a prebuilt ABACUS from these images should instead build from source (docs/quick_start) or use a release binary.
Reminder
AGENTS.mdanddocs/developers_guide/agent_governance.md.source/changes.Linked Issue
Fix #
Unit Tests and/or Case Tests for my changes
What's changed?
Governance Notes